Expands InternalExportDataProvider per dataverse-spi 2.1.0 and refactors the DDI exporter to utilize the new functionality#11799
Expands InternalExportDataProvider per dataverse-spi 2.1.0 and refactors the DDI exporter to utilize the new functionality#11799landreev wants to merge 31 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
|
Note that the PR now builds under Jenkins (with a snapshot release of dataverse-spi that is now published). |
This comment has been minimized.
This comment has been minimized.
…he data objects are retrieved. #11405
Resolved the following merge conflicts: modules/dataverse-parent/pom.xml src/main/java/edu/harvard/iq/dataverse/harvest/server/OAIRecordServiceBean.java src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java
... into IQSS/11405-refactor-exports (the working branch behind an already existing PR). merging without further ado.
…ing as is when committed, since there was only one dataset and one datasetversion in the throwaway database. #11405
…ularly-scheduled workflow action integration tests. #11405
Resolved conflicts: src/main/java/edu/harvard/iq/dataverse/api/Metadata.java (seemingly trivial conflict, on account of a new API parameter added in this PR vs. extra annotations having been added in the develop branch). #11405
| assertTrue(false, "Unexpected XMLStreamException when attempting to parse the dataDscr section: " + xse.getMessage()); | ||
| } finally { | ||
| if (xmlr != null) { | ||
| try {xmlr.close();} catch (XMLStreamException e) {} |
There was a problem hiding this comment.
🚫 [reviewdog] <com.puppycrawl.tools.checkstyle.checks.blocks.LeftCurlyCheck> reported by reviewdog 🐶
'{' at column 21 should have line break after.
There was a problem hiding this comment.
Don't tell me what to do, reviewdog. That was formatted like that on purpose, to emphasize the fact that we don't care about that particular exception at that stage of the test.
This comment has been minimized.
This comment has been minimized.
|
@poikilotherm Hi, I'm looking at the test results above, and seeing this failure to "Release Snapshot" for Dataverse SPI: Am I correct assuming this is an old action that simply needs to be removed, now that dataverse-spi lives in https://github.com/gdcc/dataverse-spi? |
|
@landreev Yes, the in-tree SPI stuff can be completely eradicated. ☢️ 😄 |
…hot releases of it from the source tree; since dataverse-spi has been living under https://github.com/gdcc/dataverse-spi for some time now. #11405
…led on the pr on account of it. #11405
|
📦 Pushed preview images as 🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name. |
| // DO in fact contain enough information to generate the otherMat sections | ||
| // properly, whether this is a short or a full version of the DDI. I am however leaving | ||
| // this method here for reference. | ||
|
|
There was a problem hiding this comment.
As per your comment, IMO this method should be tagged @deprecated to indicate its imminent removal.
| * @varQuantityMap mapping of ordered datatables -> number of variables in each | ||
| * @exportDataProvider data provider instance | ||
| * @variablesBatchSize number of variables to use in forming processing batches | ||
| */ |
There was a problem hiding this comment.
The integration test should be in the same package as the class (it already is in ...export, so it should move to ...export.ddi, the package this class is a part of). Then the method can be made package private and is accessible to the test right away without exposing it.
| logger.fine("total number of variables in this batch: " + varQuantityThisBatch); | ||
|
|
||
| int count = 0; | ||
| //for (int i = 0; i < tabularFileDetails.size(); i++) { |
There was a problem hiding this comment.
Is this a comment to explain the iterator pattern? If so, please use human language and remove the code block ;-)
|
|
||
| } | ||
|
|
||
| public void exportFormats(Dataset dataset, List<String> formatNames) throws ExportException { |
There was a problem hiding this comment.
This method is missing Javadocs explaining scope, reasoning and behavior.
Also:
- I encourage you to use an empty list instead of a null value, which IMO is much more Javaish.
- There is no null check on the dataset parameter.
- Why not use DatasetVersion and split the logic in two methods? The getExport() method uses a DatasetVersion, thus IMO using a DatasetVersion + list of formats is more aligned. There can be a second method with Dataset + list of formats for convenience, that handles the business logic of which version may be exported (calling the one with version + formats internally).
For Johannes I will be adding a method with cardinality in the other direction: one format, multiple versions. Eventually, we will refactor the service to support exporting arbitrary versions, where it also makes sense to have version + formats.
| Set<DatasetMetadataPredicates> predicates = query.getDatasetPredicates(); | ||
|
|
||
| for (DatasetMetadataPredicates p : predicates) { | ||
| // @todo This is pending on adding a dedicated DATASET_LEVEL_ONLY predicate |
There was a problem hiding this comment.
No. The DatasetExportQuery has predicates only for the dataset's own metadata:
Predicates for controlling which dataset metadata is included in an export and at what level of detail.
The dataset export query may have a distinct FileExportQuery containing detailed information as its own predicates about the file metadata. At the moment, there is only one case where this method may return false:
- There is a file export query within the dataset export query and it has a predicate "SKIP_FILES". (Otherwise it's either all files, public and/or tabular only).
We might need to tighten the definition in the export query spec, here's what it says now:
File metadata shaping is optional: if no FileExportQuery is provided, methods that include file metadata will apply their own defaults. Methods that do not return file metadata will ignore any nested FileExportQuery.
At time of writing, this seemed fine. But maybe it isn't? Should we do an explicit default? Do I remember correctly that in the past getDatasetJson() did not return file level metadata? (Then we should keep this default for backward compatibility, changing it would require a breaking change - and we shouldn't do those at this point.)
| // generated, should we return it (potentially moving MUCH more json | ||
| // than the client needs, or spend extra cycles generating the short | ||
| // form from scratch? - I'm choosing to go with latter. | ||
| if (jsonRepresentationNoFiles == null) { |
There was a problem hiding this comment.
My gut tells me: if we really want to cache here, let's use a thread-safe map and the query as the key. In the edge case that someone really wants to retrieve the metadata again, they get a consistent result, as the query might be different the next time around!
| dataset.setLastExportTime(null); | ||
| } | ||
|
|
||
| public void clearCachedFormats(Dataset dataset, List<String> formatNames) throws IOException { |
There was a problem hiding this comment.
The behavior here is not aligned with exportFormats. With that method using a null list will clear all, here you don't even check for null, risking an NPE.
I'd argue the same way as exportFormats: use a List, either empty or filled, disallow null. If empty, clear all.
| } | ||
|
|
||
| dataset.setLastExportTime(null); | ||
| public void clearAllCachedFormats(Dataset dataset) throws IOException { |
There was a problem hiding this comment.
This is a redundant method. People can just call the other defined methods. If you really want this for convenience, don't add a second implementation.
| @@ -1 +1 @@ | |||
| DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT,FilesIT,SearchIT,InReviewWorkflowIT,HarvestingServerIT,HarvestingClientsIT,MoveIT,MakeDataCountApiIT,FileTypeDetectionIT,EditDDIIT,ExternalToolsIT,AccessIT,DuplicateFilesIT,DownloadFilesIT,LinkIT,DeleteUsersIT,DeactivateUsersIT,AuxiliaryFilesIT,InvalidCharactersIT,LicensesIT,NotificationsIT,BagIT,MetadataBlocksIT,NetcdfIT,SignpostingIT,FitsIT,LogoutIT,DataRetrieverApiIT,ProvIT,S3AccessIT,OpenApiIT,InfoIT,DatasetFieldsIT,SavedSearchIT,DatasetTypesIT,DataverseFeaturedItemsIT,SendFeedbackApiIT,CustomizationIT,JsonLDExportIT,WorkflowsIT,LDNInboxIT,LocalContextsIT,CorsIT,GuestbooksIT,LocallyFairIT | |||
| DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT,FilesIT,SearchIT,InReviewWorkflowIT,HarvestingServerIT,HarvestingClientsIT,MoveIT,MakeDataCountApiIT,FileTypeDetectionIT,EditDDIIT,ExternalToolsIT,AccessIT,DuplicateFilesIT,DownloadFilesIT,LinkIT,DeleteUsersIT,DeactivateUsersIT,AuxiliaryFilesIT,InvalidCharactersIT,LicensesIT,NotificationsIT,BagIT,MetadataBlocksIT,NetcdfIT,SignpostingIT,FitsIT,LogoutIT,DataRetrieverApiIT,ProvIT,S3AccessIT,OpenApiIT,InfoIT,DatasetFieldsIT,SavedSearchIT,DatasetTypesIT,DataverseFeaturedItemsIT,SendFeedbackApiIT,CustomizationIT,JsonLDExportIT,WorkflowsIT,LDNInboxIT,LocalContextsIT,CorsIT,GuestbooksIT,LocallyFairIT,TabularDataExportIT | |||
There was a problem hiding this comment.
Please do not add this test here. It's an integration test, not an API test. For historic reasons, this file has been around and collects these API tests. It should be torched with fire.
Your performance test will be automatically picked up for execution during mvn verify -Dit.groups=performance
If you feel we should make performance tests a part of the regular Github Action flow, let's add it there! (currently only integration and migration are picked up, see pom.xml property it.groups)
| datasetService.reExportDatasetAsync(dataset); | ||
| List<String> formatNames = null; | ||
| if (formats != null) { | ||
| formatNames = new ArrayList<>(Arrays.asList(formats.split(","))); |
There was a problem hiding this comment.
As user input, the list of formats shall be verified on this code level, before entering any services.
Side note: The try/catch thing around everything is an anti-pattern and always has been. Let's not fix it here, but I wanted to repeat this, as it is an anti pattern all over our API code.
|
Sorry, I submitted the review before adding a comment! My bad! Overall, I think the SPI refactoring was a good choice. The code here is much clearer to read now, as responsibilities are clearer. |
Un-drafting the PR now!
What this PR does / why we need it:
See issue #11405. Short version: the current export data framework is prohibitively expensive for datasets with very large amounts of tabular data (i.e., "ingested" DataFiles that have associated DataTable entities , with large numbers of child DataVariable objects). This PR improves it, via adding code that retrieves the data in manageable chunks. This in turn relies on the new version of the
ExportDataProviderinterface from https://github.com/gdcc/dataverse-spi.Which issue(s) this PR closes:
Special notes for your reviewer:
Note that the PR builds with a snapshot release of dataverse-spi:
I.e., the version 2.1.0 still needs to be properly released under gdcc. But I don't think that should be stopping putting the PR into review and merging it rather sooner than later.
Note that this PR also removes
modules/dataverse-spi, and the action that used to build snapshot releases of it (.github/workflows/spi_release.yml) from the main source tree; now that dataverse-spi lives under gdcc.Note specifically the test for the functionality,
TabularDataExportITadded in this PR. I have added it to the list of integration tests that are performed automatically by the workflow action (intests/integration-tests.txt) even though, strictly speaking, it is not a real integration test. I.e., it does not rely on calling a fully-functional Dataverse instance started remotely. It relies instead on the new test framework added in @poikilotherm's #12365 where a local Docker instance of the database is spun up in which some dummy objects - Datasets, Versions, Datatables and Variables etc are created instantly, allowing testing of complex functionality that requires real database access. Without this new framework, the only way to test an export of a dataset with N large tabular files would be to actually ingest these N files via the API in the remote Dataverse instance used by the RestAssured tests. Which would add to the ongoing bloating of the test suite, and still would not allow to test the underlying functionality as thoroughly (for example, there is no easy way to count the actual database queries under such scenario).You can run the test in question in your own dev. environment via
mvn test -Dtest=TabularDataExportIT#exportTabularMetadataIf you do not have Docker running, the test will be politely skipped. Otherwise it will create a throwaway postgres instance and proceed to test tabular data exports and associated data retrieval, comparing the results produced using the legacy methods vs. the new and optimized versions added in this PR and more.
Suggestions on how to test this:
Please see the info under the "notes for the reviewer" and feel free to play and experiment with the new test described there. Please keep in mind that the test is still somewhat of a reference demonstrator of the new
JpaEntityManagerService-based framework from #12365. It can be expanded and improved going forward.The ultimate test would be to test before vs. after on some real-life monster datasets.
I would suggest these bad boys on qa:
https://qa.dataverse.org/dataset.xhtml?persistentId=doi:10.7910/DVN/KZW8R2 (207 ingested tab. files; 186906 variables total)
https://qa.dataverse.org/dataset.xhtml?persistentId=doi:10.7910/DVN/S5BRBO (134 tab. files; 536006 variables).
"ddi" (aka the "full" ddi) is the export format that needs to be tested.
Note that some extra steps will be required in order to test this on qa:
export_ddi.cachedwill need to be erased between testing runs.Happy to assist with the above if needed.
Does this PR introduce a user interface change? If mockups are available, please link/include them here:
Is there a release notes update needed for this change?:
Additional documentation: